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Abstract 

There is a well known construction of weakly continuous valuations 
on convex compact polytopes in W 1 . In this paper we investigate when 
a special case of this construction gives a valuation which extends by 
continuity in the Hausdorff metric to all convex compact subsets of 
R n . It is shown that there is a necessary condition on the initial data 
for such an extension. In the case of M 3 more explicit results are 
obtained. 
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Introduction. 



The aim of this paper is to discuss a well known construction of valuations 
on convex compact polytopes in M. n and study the question when the con- 
structed valuations can be extended by continuity in the Hausdorff metric 
from polytopes to the class of all convex compact sets. We show that in 
general there is a non-trivial obstruction to such an extension. The case of 
3-dimensional space is studied in greater detail, and we obtain necessary and 
sufficient condition on the initial data of the construction under which the 
valuations do extend by continuity to all convex compact sets. The main 
results of the paper are Theorems 10.51 10.7} 10.101 below. 

Let V be a finite dimensional real vector space, n = diml^. Let V(V) 
denote the family of all convex compact polytopes in V, and let 1C(V) denote 
the family of all convex compact subsets in V. 

0.1 Definition. A valuation on V(V) (resp. K,(V)) is a functional 

<f>:V(V) 

( resp. : K(V) — >■ C) 

which is additive in the following sense: for any A,B 6 V(V) (resp. K{V)) 
such that the union A U B is also convex, one has 

<j){A UB)= <j){A) + <j)(B) - <f)(A n B). 

As a generalization of the notion of measure, valuations on convex sets 
have long played an important role in geometry. Since the middle 1990's 
we have witnessed a breakthrough in the structure theory of valuations |27j . 
[M],[I]-[2] 5 [TT]-[T3] , [22] -[32], [38] ■ This progress in turn has led to immense 
advances in the integral geometry of isotropic (in particular, Hermitian) 
spaces (see [2],[6],[ll],[T5]) and an understanding why certain classical no- 
tions from geometric analysis are indeed fundamental (see [22] , [31] , [29J , [30J , [37] ) . 

0.2 Definition. A valuation <fi on JC(V) is called continuous if is continuous 
in the Hausdorff metric on JC(V). 

The notion of weak continuity of a valuation is often more appropriate for 
polytopes; it is due to Hadwiger [23]. Let us recall it. Fix £ := {£i, • • • , £ 8 } C 
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V* an s-tuple of linear functional on V. Let V% denote the family of poly- 
topes in V(V) of the form 

P^y) = {xeV\C i (x)<y l \ft = l,...,s} 

where y = (yi, . . . ,y s ) E W are such that P^(y) are compact. 

0.3 Definition. A valuation on V(V) is called weakly continuous if for 
any £ the function 

y ^ cj>{P t {y)) 

is continuous with respect to y in the set where P%(y) is compact. 

Clearly the restriction to V(V) of a continuous valuation on /C(V) is 
weakly continuous. The first question studied in this article is the converse 
one: when does a weakly continuous valuation on V(V) extend to a contin- 
uous valuation on ]C(V)1 Clearly this extension is unique if it exists. Let 
us also mention that the problem when a valuation on polytopes extends by 
continuity to valuations on fC(V) was discussed in a recent paper [3B] in a 
somewhat different situation. 

Weakly continuous translation invariant valuations on V(V) were de- 
scribed completely by P. McMullen [33]. All such valuations are obtained 
by a general relatively explicit construction. Let us describe a particular 
case of it which will be studied in this article in greater detail. 

Let us fix a Euclidean metric on V. Let us denote by GkiV) the manifold 
of pairs (E, I) where E C V is a k- dimensional linear subspace of V, and I is 
a line orthogonal to E. 

Let / : GkiV) — > C be a continuous function. For a polytope P G V(V) 
let us denote by J~k{P) the set of fc-faces of P. For any /c-face F G Fk{P) let 
us denote by F the (only) fc-dimensional linear subspace of V containing a 
translate of F to 0. Let us denote by 7^ the exterior angle of P at F, that is 
7i? is the subset of the unit sphere 5'(F- L ) of F 1 - consisting of all n G 5'(F- L ) 
such that for some (equivalently, any) x belonging to the relative interior of 
F the scalar product (n,p — x) < for any p G P. 

Finally for P G V(V) define 
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where dl denotes the Lebesgue measure on the sphere S^F- 1 ) such that the 
total measure of S'(F- L ) is equal to 1 (here we consider f(F, •) as an even 
function on the unit sphere S^F -1 )). 

By the (easy part of the) P. McMullen's theorem [33J, 0/ is a translation 
invariant even weakly continuous valuation on V(V). 

0.4 Example. Take / = 1. Then 0/ is proportional to the fc-th intrinsic 
volume Vk (see the book [35] for this notion). 

The goal of this paper is to study the following question: when does the 
valuation <pf extend by continuity in the Hausdorff metric to fC(V)7 It turns 
out that such an extension does not always exist. Notice that by a result of 
Groemer [2TJ if 0/ extends by continuity to )C(V) then this extension will be 
automatically a valuation on JC(V). In this case we will denote this extension 
by the same symbol (/>/. 

Our first main result is as follows. 

0.5 Theorem. Let n := dimK > 3 and 1 < k < n — 2. There exists a real 
analytic function f: Gk{V) — > C such that 0/ does not extend to 1C(V) by 
continuity. 

0.6 Remark. (1) It is well known that for k = n — 1 the valuation <pf does 
extend by continuity to fC{V) for any continuous function /. 

(2) Obviously the simplest case when the assumptions of Theorem l0.5l are 
satisfied is n = 3, k — 1. It turns out that in this case one has a much more 
precise statement, see Theorem 10. 101 below. 

(3) We would like to emphasize the following phenomenon: the class of 
smoothness of the function f is not really important, it can be real analytic, 
and in fact even 0(n)-finite|j 

(4) By Proposition 10.91 below, the subset of the space C°°(G k (V)) of 
infinitely smooth functions / such that <pf admits an extension to JC(V) by 
continuity is a closed linear subspace. 

(5) In a very recent preprint [25] Theorem 10.51 was proved by a very 
different method. 

1 Recall the definition of 0(n)-finite vectors in a representation of 0(n) in a vector 
space U: a vector £ £ U is called 0(n)-finite if it is contained in a finite dimensional 0(n)- 
invariant linear subspace. It is well known that if U is the space of (smooth, continuous, 
or L 2 ) functions on a homogeneous space of 0(n), then all 0(n)-fmite functions are real 
analytic. 
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Now we are going to explain some more precise forms of Theorem 10.51 In 
the rest of the paper we will always assume that / is infinitely smooth. 

Let us denote by C°°(X) the space of infinitely smooth functions on a 
manifold X. Let us denote by Grk(V) the Grassmannian of fc-dimensional 
linear subspaces of V. Let us define the linear map 



where P(£^ ± ) denotes the set of lines in E 1 - (i.e. the projectivization of E- 1 ), 
dl is the normalized Lebesgue measure on P(£ ,_L ). Clearly the map S is onto. 
Here is our second main result. 

0.7 Theorem. Let n := dimV^ > 4 and 2 < k < n — 2. There exists a 
closed proper subspace of C cc '{Grk{y)) with the following property: if f £ 
C°°(Gk(y)) is such that <pf extends by continuity to JC{V), then S(f) belongs 
to this closed proper subspace. This implies that the subset of f G C°°(Gfc(V)) 
such that <pf extends by continuity to IC(V) is contained in a closed proper 
linear subspace. 

0.8 Remark. This theorem already implies Theorem 10.51 for 2 < k < n — 2. 
Actually, as it will be seen from the proof, the closed proper subspace of 
C°°(Grk(V)) mentioned in the theorem is equal to the image of the so called 
cosine transform on the Grassmannian Grk(V). For 2 < k < n — 2 it is known 
to be a proper subspace by a result of Goodey, Howard, and Reeder [ISj ; see 
also [5] for a more precise description of this subspace. The proof of the 
proposition uses the Klain imbedding [28J of even valuations to functions on 
the Grassmannian and the result by J. Bernstein and the author [8j stating 
that the image of the Klain imbedding is equal to the range of the cosine 
transform. This implies also that the orthogonal complement of the range of 
the cosine transform contains 0(n)-finite functions, which are known to be 
real analytic. 

The case k — 1 is not covered by Theorem 10 .7| here the situation is more 
subtle since the range of the cosine transform on C oc (Gri(V)) is equal to 
the whole space. Here we will (easily) see that in order to prove Theorem 
10.51 it suffices to do that for dim^/ = 3. In this case we have a much more 



S: C^iGkiV)) 



C°°(Gr fc (y)) 



by 
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precise statement. In order to formulate it, it will be necessary to rewrite 
the construction of 0/ in a more invariant way without using explicitly the 
Euclidean metric on V. Then the group GL n (M) will act on all spaces. It 
will be necessary to consider / not as a function on GkiV) but as a section 
of a certain GL n (M)-equivariant line bundle over GkiV). 

Let us describe this. We will do it for dim V > 3 and any 1 < k < n — 2. 
First in metric free terms we identify Gk{V) with the set of pairs (E, I) where 
E £ Grk(V) is a k- dimensional linear subspace, and I is a line contained in 
the annihilator E C V*. 

Let p : GkiV) — > Grk(V) denote the projection p(E, I) = E. Let L — > 
Grk{V) be the line bundle whose fiber over E £ Gr k {V) is equal to the space 
of Lebesgue measures on E. Note that a choice of a Euclidean metric on V 
defines a trivialization of L. 

Next let us denote by \uJa k /Gr k \ the line bundle over GkiV) of relative 
densities of GkiV) over GrkiV). Let us recall its definition. Let q: X — > Y 
be a smooth map between smooth manifolds. Assume that q is a submer- 
sion, i.e. its differential at every point is onto. (In our case X = Gk{V), Y = 
Grk(V),q = p.) The fiber over a point x £ X of the line bundle of rela- 
tive densities I^x/yI is, by definition, equal to the (1-dimensional) space of 
Lebesgue measures on the tangent space at x to the fiber p~ l {p(x)). If q is 
proper, then we have a canonical map C°°(X, |(X>x/y|) — > C°°(Y) given by 
integration along the fibers of p. 

Note also that a choice of a Euclidean metric on V defines a trivialization 
of \u)G k /Gr k I (indeed a Euclidean metric on V defines a Haar measure on each 
fiber of p) . Let 

M := p*L ® \u Gh/Grh \. 

Thus a choice of a Euclidean metric on V defines a trivialization of M, and 
hence an isomorphism 

C 00 (Gk(V),M)~C °(G k (V)). (0.1) 

In this notation and with the isomorphism (10.11) . the above construction of 
(f) f can be rewritten as follows. Let / £ C°°(G' fc (V r ), M). Let P £ V(V), F £ 

JFk{P). Then f is a Lebesgue measure on F. @ Hence f F (^f jF fj is a 

2 To be more precise, 7f is a subset of the manifold ¥ + (E ± ) of oriented lines in I? , 
and here we integrate / over the image of -fF in P(£ ,_L ) under the natural two sheeted 
covering P + (E ± ) — > P(£ ,_L ) given by forgetting the orientation. 
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number. Then 




(0.2) 



Set 

S := {/ G C°°(Gk(V), M)\<j>f extends by continuity to K,{V)}. 

Thus under the identification (10.11) the subspace H is isomorphic to the space 
of smooth functions / on Gk(V) such that 0/ extends by continuity to /C(V). 
In Proposition 11.11 we will show 

0.9 Proposition. S is a closed GL(V) -invariant linear subspace ofC°°(Gk{V), M). 

The next main result of this paper is the following one; later on it will be 
formulated more explicitly. 

0.10 Theorem. Let n = 3, k = 1. The natural representation of GLs(M) in 
C°°(Gi(V), M) has length three. The (closed GL 3 (R) -invariant) subspace H 
has length two. Moreover H coincides with the image of an explicit GL 3 (R)- 
equivariant map closely related to the construction of valuations using inte- 
gration with respect to the normal cycle (described below). 

0.11 Remark. (1) As stated, Theorem 10. 101 does not provide a unique rep- 
resentation theoretical characterization of the subspace H. However with a 
little bit of extra work it can be shown that H is the only GL3(M)-equivariant 
closed subspace of C°°(Gi(V), M) of length two whose irreducible subquo- 
tients have the minimal (and equal to each other) Gelfand-Kirillov dimension 
among the three irreducible subquotients of the whole space (the third ir- 
reducible subquotient has strictly larger Gelfand-Kirillov dimension). We 
will not pursue this charaterization in this paper, but people familiar with 
the Gelfand-Kirillov dimension can easily deduce the above characterization 
from our arguments. 



(2) For any n and k the recent preprint [23] contains a sufficient condition 

for a section to belong to H (see Theorem 4.1 there). 

The proof of this theorem uses the Klain imbedding theorem, construc- 
tion of valuations using integration with respect to the normal cycle, and a 
detailed study of the representation of GL 3 (R) in C°°(Gi(E 3 ), M); the latter 
step uses the Beilinson-Bernstein localization theorem. 
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The paper is organized as follows. In Section [TJ we prove the main results 
on valuations except the case n — 3, k — 1. In Section [2] we prove the case 
n = 3, k — 1 modulo some representation theoretical computations which 
are postponed to Section [U In Section [3] we remind some representation 
theoretical background; that section does not contain new results. In Section 
H]we make the computations with representations of GL 3 (M.); the main result 
of this section is Theorem 14.11 

Acknowledgements. I am very grateful to S. Sahi who has pointed 
out a mistake in an earlier version of the paper. I thank F. Schuster for his 
numerous remarks on the first version of the paper. I thank D. Gourevitch, 
D. Hug, and W. Weil for useful discussions. 

1 Proof of main results on valuations when 

n > 3. 

We keep the notation of the introduction. 

1.1 Proposition. The space H is a closed GL(V) -invariant linear subspace 
ofC°°(G k (V),M). 

The linearity and GL(V)-invariance of S are obvious. In order to prove 
that S us closed we will need some preparations. 

Let us denote by Val^iy) the space of translation invariant continuous 
even /c-homogeneous valuations on fC{V). Equipped with the topology of 
uniform convergence on compact subsets of fC(V), Val^iV) is a Banach 
space. Let us recall the construction of the Klain imbedding 

K : Vall v (V) m> C(Gr k ,L) 

which is a continuous linear map commuting with the natural action of 
GL(V). Let G Val^iV). For any subspace E e Gr k {V) the restric- 
tion of 0| e is a k- homogeneous valuation. By a result of Hadwiger [21], (/)\e 
is a Lebesgue measure on E. Let us denote by K the map h->- [E (->■ 4>\e]- 
Thus K : Valfiy) — ► C{Gr k {V),L). A deep result due to D. Klain [2Z] 
(heavily based on [2S]) says that K is injective. 
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Let us denote by Val e k ' sm (V) the subspace of smooth k- homogeneous even 
valuations (a valuation <fi G Val(V) is called smooth if the map GL(V) — > 
ValiV) given by g \-> g(<p) is C°°-differentiable. Val e k v,sm (V) has a natural 
structure of a Frechet space. Moreover 

K : Val e k v ' sm (V) ^ C°°(Gr k (V), L). 

The Casselman-Wallach theorem (see Theorem 13. 91 below) implies the follow- 
ing claim. 

1.2 Claim. The Klain imbedding 

K : Val e k v ' sm (V) ^ C°°{Gr k {V), L) 

has a closed image and induces an isomorphism ofVal e k ,sm (V) onto its image 
as topological vector spaces. 

Let us denote by T the map H — > Val^iV) given by / !->■ </>/. Note also 
that we have the canonical map 

S : C°°(Gk(V), M) — > C°°(Gr k {V), L) (1.1) 

given by the integration along the fibers of the projection p : Gk{V) — > 
GrkiV)- The map S was defined in the introduction under a choice of Eu- 
clidean metric. To rewrite it in invariant terms let us recall that by def- 
inition M = p*L ® u Gk / Grk . Then for any / G C°°(G k (V), M) and any 
E G Gr k (V), the restriction f\ p -i(E) is a section over p~ l (E) of the line bun- 
dle |cOp-i(£)| ® L\e- Hence one has f p -ir E ^ f G L\e- Now it is clear that S is 
a continuous linear map commuting with the natural action of GL(V). 
The next lemma is obvious. 

1.3 Lemma. The composition 

K oT :E — > C(Gr k , L) 

is equal to S\s- 

1.4 Lemma. For any f G H the valuation <pf is smooth, i.e. 

T{E) C Val e k v ' sm (V). 
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Proof. By Lemma O for any f e E, K(4> f ) G C°°(Gr k (V), L). Let us 
show that this implies that 0/ is smooth. By Claim [TT2"| K(Val'j?' sm (V)) is a 
closed subspace of C°°(Gr k (V), L). Its closure K(^a/f' sm (\/)) in C(Gr fc (V), L) 
contains KiVal^iV)). It is easy to see that 

K(Val e k v ' sm (V)) n C°°(Gr fc (\/), L) = >sm (V)). 

Hence there exists ip G Val^ ,sm (V) such that if(V0 = K(<t>f)- Since K is 
injective on Valf(V) we get ^ = 0/. Q.E.D. 

Proof of Proposition ll.il It remains to show that E is a closed subspace of 
C7 co (G? fc (V r ), M). First let us show that the map T : E — > Val e k v ' sm (V) is con- 
tinuous when E is equipped with the topology induced from C°°(Gk(V), M). 
By Claim 11.21 it is enough to show that K o T is continuous. By Lemma 11.31 
K o T = S which is obviously continuous. 

Now let us assume / G H. Let {/at} C E be a sequence such that 
/tv — > f ■ The sequence {T/tv} C Val e k v ' sm (V) is a Cauchy sequence due to 
the continuity of T. Hence there exists G Val e k v,srn {V) such that T/jv = 
/jv — >■ 0. Hence (K o T)(/ iV ) — >■ #(0)- Also — ► Sf. Hence 
5/ = K((p). By construction, it is also clear that for any polytope P G V(V) 
one has /jv (P) — > 0/(P)- But also /jv (P) — > 0(P). Hence 0/(P) = 0(P) 
for any P G P(V r ). This means that 0/ admits an extension by continuity to 
lOy). Hence / G H. Q.E.D. 

1.5 Theorem. 

T(E) = Vall v ' sm (V). 

Proof. Recall that T : E — > Val e k v,sm {y) is a continuous GL(V)- 
equivariant map. Moreover it is non-zero by Example 10.41 By the Irre- 
ducibility Theorem [TJ the space Val e k ' sm {V) is GL(V)-irreducible. Hence 
ImT is dense in Val™' sm (V). By Proposition 11.11 E is a closed subspace of 
C°°(Gk{V), M). Hence by the Casselman-Wallach theorem (see Theorem l3.9l 
below) ImT is closed. Hence ImT = Val e k ' sm {V). Q.E.D. 

1.6 Theorem. The subspace S(E) C C°°(Grk(V), L) coincides with the 
range of the cosine transform. 

Proof. Recall that by Lemma O S\ s = K o T. Thus S(E) — (K o 
T)(E) = K(V al e k ' sm (V)) where the last equality follows from Theorem 11.51 
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But K(Val e k VySm (V)) coincides with the range of the cosine transform by [8]. 
Q.E.D. 

Theorem 11.61 immediately implies Theorem 10.71 
1.7 Lemma. Let n > 3, 1 < k < n — 1. Then 

E n KerS ^ KerS. 

Proof. Let us fix a Euclidean metric on V, and let us identify C°°(Gk{V), M) 
with C°°(Gk{V)). In order to prove the lemma it is enough to construct a 
function / G KerS and a simplex A such that 0/(A) ^ 0. Let us fix an ar- 
bitrary n-simplex A. Let Fi,...,F t be all of its k- faces (then t = (^i))- 
Let A; := p-^Fi) C G fc (V). Thus each A { is isomorphic to HF 1 ^*- 1 . 
Let us choose an arbitrary function g G C°°((jfc(V)) such that = 
for i > 1, p 7^ 0, and J^f? = 0. Then clearly ff (A) 7^ 0. Let 

:= S(g) G ^(Gr^y)). Let f := g - p*g. Then clearly / G KerS, 
f\ A . = g\ A . for any i = l,2,...,t, and hence 0/(A) = <fi g (A) ^ 0. Hence 
/ ^ H. This proves the lemma. Q.E.D. 

So far we have proven Theorem 10.71 It implies Theorem 10.51 in all cases 
except when k = 1 and n > 3. Let us deduce the case k — 1, n > 3 from the 
case fc = l,n = 3. Let be an n-dimensional vector space, n > 3. Let us 
fix a 3-dimensional subspace W C V. Let / G C°°(Gi(V^)) be an arbitrary 
smooth function. Let 4>f be the corresponding valuation on polytopes V(V). 
Let us describe explicitly the restriction of <pf to V{W) C V(V) in terms 
intrinsic to W. We will construct a function g G C°°(Gi(W)) such that the 
restriction of (fij to V(W) is equal to g . 

The function (7 can be described as follows. Let p w ± : V — > W 1 - denote 
the orthogonal projection to W x . Then 

g(E,l) = Kn f f(E,m)y/l-\p w ±(m)\*dm, (1.2) 

where the integration is over the unit sphere of the space I © W- 1 , dm is 
rotation invariant Haar measure on this sphere, | • | under the square root is 
the Euclidean norm, and n n 7^ is a normalizing constant depending on n 
only. We leave it to the reader to verify this elementary formula. 
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Clearly the map / h-> g defined by ( 11.21) is a continuous linear operator 



U: C 0O (G 1 (F)) — >• C°°(Gi(W)). 

It is easy to see that this map is onto. 

Let g be an 0(3)-finite function on G\{W) such that the valuation <p g 
does not extend by continuity to all convex compact subsets of W; for the 
moment we just assume its existence. It suffices to show that there exists an 
0(n)-finite function / on G\(V) such that U(f) = g. Then we would have 
that the restriction of <pf to V(W) is equal to g ; this implies that <f)f does 
not extend by continuity to all convex compact subsets of V. 

Notice that existence of such an infinitely smooth function / follows from 
the surjectivity of U on infinitely smooth functions. Next we have the obvious 
imbedding 0(3) C 0(n). The map U commutes with the action of 0(3) on 
both spaces. Clearly any 0(n)-finite function is 0(3)-finite. It follows that 
since 0(n)-finite functions are dense in C°°(Gi(V)), their image under T is 
dense in C°°(Gi(W)), and any function in this image is 0(3)-finite. But any 
dense linear 0(3)-invariant subspace of C°°(Gx(W)) consisting of 0(3)-finite 
functions is equal to the space of all 0(3)-finite functions on C°°(Gi(V)); 
this easily follows from the fact that the natural representation of 0(3) in 
functions on Gi(W) has finite multiplicities due to transitivity of the action 
on Gi(W). This finishes the proof of Theorem 10.51 



2 The case n = 3, k = 1. 

The main result of this section, Theorem 12.5} is stated below. Let us start 
with some technical preparations. 

Let us study the natural representation of the group GL(V) in the space 
C°°(Gk(y), M). Let us denote by P the parabolic subgroup of GL(V) with 
the Levi component GL k x GL n _k-i x GL\. Let x '■ P — * C* be the character 
of P given by 

A * * \ 

OB* = IdetAI" 1 • |det5| • |0|" (n ^ 1} 
C \) 

where A e GL k , B e GL n _ k _ x , C e GL 1 . 

The following proposition is straightforward. 
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2.1 Proposition. The natural representation ofGL{V) in the space C oa {Gkiy)-, M) 
is equal to the induced representation Ind^x (the induction is not unitary). 

Let us study in greater detail the case n — 3, k = 1. Then the corre- 
sponding parabolic subgroup P is the subgroup of upper triangular matrices 
of size 3, and 



In Section H] below we will prove the following representation theoretical 
result. 

2.2 Proposition. Let n = 3, k = 1. The representation IndpX is of length 
3 as an GL(V) -representation. Moreover the irreducible subquotients are 
pairwise non-isomorphic. 

Recall from Section [TJ that we have the map (integration along the fibers) 



We will need a lemma. 

2.3 Lemma. KerS has length 2 as GL(V) -module. 

Proof. By Theorem 11.61 the subspace S(E) C C 00 (Gri(V) , L) coincides 
with the range of the cosine transform. But in this case the cosine transform 
is an isomorphism (see e.g. [39]). Moreover the source and the target spaces 
of the cosine transform are irreducible GL(V)-representations (this is well 
known; see e.g. [26]). Hence Im(S) = C°°(Gri(V), L) is irreducible. This 
and Proposition 12.21 imply that length of Ker(S) is equal to 2. Q.E.D. 

Let us remind an important construction of continuous valuations called 
integration with respect to the normal cycle; it will be necessary for formula- 
tion of the main result of this section. For a convex compact subset K C V 
and a point x £ K let us define a tangent cone T X K to K at x: 




(2.1) 



S: C°°{GuM) 



C°°(G ri (V),L). 



T X K := {v £ V\ x + tv £ K for < t < 1}. 
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Clearly T X K is a convex cone in V. Define the dual cone 

(T X K)° := {£ G V*\ <£,v><0 for all v G T X K}. 

Let us denote by P + (V*) the oriented projectivization of V*, namely the set 
of oriented one dimensional linear subspaces of V*. Another way to view 
it is as the quotient space (1 / *\{0})/M > o, where M> acts of V* by usual 
multiplication. 

Now let us define the normal cycle N(K) as a subset of P + (V*) x V: 

N(K) := U xeK (T x K)°/R >o- 

It is well known that N(K) is a compact set of Hausdorff dimension n — 1 
(n = dim V). It is easy to see that an orientation of V induces an orientation 
of N{K). 

To construct a continuous valuation, let us fix 

u G C°°(F + (V*), n n - v ) <g> or(V), 

where stays for the vector bundle whose sections are n — 1 forms, and 
or(V) is the orientation line of V. Then define a functional on K,(V): 

I u. 

JN(K) 

It is well known (see e.g ??) that this functional is a continuous valuation 
on /C(V). It is clear that if cu is translation invariant with respect to V then 
the corresponding valuation is translation invariant. The space of translation 
invariant w's can be written 

©^ 1 c oo (p+(y*), ir-^- 1 ) ® aV* ® or(y). 

Elements of C ,00 (fi"" 1 ~ fe (P + (\/*)) ® A fc V* ® or(V)) define /c- homogeneous 
valuations. Even valuations are determined by even sections of the latter 
space, when the parity on them is induced by the involution on F + (V*) of 
the change of orientation. This space of even section over F + (V*) can be 
view as the space of all sections of an appropriate vector bundle over the 
projective space P(V*) of non-oriented lines in V*. It is not hard to see that 
this vector bundle is 

C °°(F(V*), Q n ~ k - 1 ® e n _ k ) ® A k V* ® or{V), 
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where £ n _fc is the trivial line bundle for n — k even, and for n — k odd its 
fiber over / G P(V^*) is equal to the orientation line or(l) of 

Let us introduce more notation. Let Wal^ denote the space of all val- 
uations on convex compact polytopes which are fc-homogeneous and even 
(without any continuity assumptions). Let 

T: C°°(G k (V),M) ^VValt 
be given by / h->- 0/ as in (10.21) . Let 

g : C°°(¥(V*), fi""^ 1 ® e n _ fc ) <g> A V* ® or(V) — >• VVal+ 

be the map u; i— > [K i— >■ J^/^ w]. The image of is contained in continuous 
valuations Val^. Clearly T ,Q are GL(K)-equivariant. 

2.4 Proposition. There exists a GL(V)-equivariant map 

H : C°°(F(V*), n^- 1 ® e n _ fc ) <8> A fc V* <8) or(V) — )• C°°(G fe (V), M) 

such that Q = Jo'H. 

Proof. Let us construct H explicitly. Fix 

uj g c°°(¥{y*), w 1 -*- 1 <g> e n _ k ) ® A k V* <g> or(V). 

Let £? G Gr h (V), and let / C E L be a line. Restriction of u to P(£ x ) C F(V*) 
is en element of C 00 (P( J B- L ), fi™-^ 1 g> e„_ fe ) <g> A k V* g> or(V). The canonical 
map V* — >■ £7* defines a linear map A fe V* — > A k E*. Hence u defines an 
element in 

C 00 (P(E ± ), n n - k - 1 g> e„_ fe ) g> A fc -B 1 <8> or(V). 
Taking the value of this section over I G P(i? _L ) we get an element in 

A n-l-k T * ¥ ( E ±j g, A fc£* or (y) g £n _ fe |^ (2.2) 

Let us show that the last space is canonically isomorphic to 

I^Gfe/GrJlj ® IrU- 

Clearly 

kvaJl, = a"- 1 -^*?^) ® or(r;p(E ± )), (2.3) 

L| B = A fe £* ®or(E). (2.4) 
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Comparing (I2.3p -( 12~^P with (I2.2p we see that we have to construct a canonical 
isomorphism 



or(T;(P(E ± )) ® or(£) = or(V) ® e n _*|,. (2.5) 

To prove this, we will use the following general canonical isomorphisms 

or(X*) = or(X), 
or(X/Y) = or(X)(g)or(Y), 
or(X ®Y) = or(X)® dimy ® or(r)^ dimX . 

Then the left hand side in (12. 5p is equal to 

or(Hom(l, E ± /l)*) <g> or(E) = 

or(l <g> (E ± /l)*) <g> ar(E) = 
or (/)®("-fc-i) g or (^)* g or (/) or (£) = 

or (/)®(«-fc) g ariy/E) ® or{E) = or{l)^ n ~ k) <g> or(V) = 

or(y) <g> e„_fc|i. 

Thus the isomorphism (12. 5 p is constructed. Hence the map % is con- 
structed. It is easy to see that it satisfies 

Q = ToU. 

Q.E.D. 



In the rest of this section we will assume that n — 3, k — 1. The following 
theorem is the main result of this section. 

2.5 Theorem. Let n = 3, k = 1. 

(1) The subspace H C C°°(G'i(l / ), M) is eona/ to i/ie image ofH. 

(2) The length of H as GL{V) -module is equal to 2, while the length of 
the whole space C°°(Gi(V), M) is equal to 3, and all irreducible subquotients 
are pairwise non-isomorphic. 

2.6 Remark. Part (1) of Theorem 12.51 provides some kind of analytic (or 
geometric) description of 5. Part (2) gives some complimentary information 
of the subspace 5: for example it may take some extra work to show that 
part (1) implies that H does not coincide with the whole space. In fact, the 
proof of part (1) uses part (2). 
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Part (2), as it is stated, does not provide a unique characterization of the 
subspace H inside C°°(Gi(V), M)). However with a little bit of extra work 
one can obtain the following slightly more precise statement: 5 is the only 
closed GL(V)-invariant subspace of length 2 whose irreducible subquotients 
have minimal Gelfand-Kirillov dimensions among irreducible subquotients of 
C°°(Gi(V), M). (In fact, one can show that the Gelfand-Kirillov dimensions 
of the three irreducible subquotients of the whole space are equal to 2,2, and 
3.) 

Proof of Theorem 12.51 First it will be useful to compute the kernel of 
%. Recall that for n = 3, k = 1 we have 

U: C 00 (P(F*),fi 1 ) ®V* ® or{V) — > C°°{Gi(y), M). 

Let us fix u E C°°(P(K*),fi 1 ) ®V* ® or(V). Clearly u E Ker(U) if and 
only if for any 1-dimensional linear subspaces E C V and I C E ± (g V*) the 
image of u vanishes under the map 

Tj*(P(F*)) <g> V* <g> or(V) — >■ T;(P(E ± )) g> ® or(V) (2.6) 

which is just the tensor product of the natural restriction maps 

T?(F(V*)) — > T^P^)), 
V"* — > E*, 

and of the identity map on or(V). 

Let us fix I C V* and describe explicitly the intersection of kernels of the 
maps (12.61) over all lines Eel' 1 . Thus we have 

t;(f(v*)) = i ® (v*/iy = i ® z x , 

(P(S ± )) = / ® (E^/l)* = l ® (l^/E). 

Then (12. 6p becomes a map 

/ g> (Z x (8) V*) <g> orV — ^ ® ((l^/E) ® £?*) <g> or(V), (2.7) 

where the map is tensor product of the identity map on the first copy of I 
and on or(K), of the canonical quotient map I 1 - — > I /E, and of the obvious 
natural map V* — > E* . 
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It suffices to describe the intersection of the kernels of maps 

l x ®V* — >(l ± /E)®E\ (2.8) 

when Z is fixed, and E runs over all lines E C l L . The map (12. 8p can be 
identified with the map 

Hom(V, I 1 -) — > Hom(E, l^/E) (2.9) 

given by composition \-t pe o o i E , where %e'- E — )■ V is the identity 
imbedding, and pe'- I 1 ' — > l L jE is the quotient map. The kernel of (12.91) 
consists of maps 0: V — > I 1 - such that <f)(E) C E. Hence the intersection 
of kernels of (I2.9P over all E C Z -1 , when L is fixed, consists of 0: V — > I 
such that any line E C I 1 - is mapped to E, or equivalently the restriction of 
to I 1 - is proportional to the identity map. Let us denote by Xi this space 
of maps tensored by Z <g> or(V). Then X\ is the intersection of the kernels of 
(12. 7p . which is the same as the intersection of kernels of (12. 6 j) over all Eel 1 ' 
with fixed Z. 

Now let us describe explicitly the quotient of T/*(P(V*)) <g> V* <g> or(V) by 
Xj. Clearly 

Xi D I <8) Hom{V/l L , Z x ) <g> or(V) = I g> (Z x ® Z) g> or(V). 
Then we have furthermore 

(t;(p(v*)) ® y* ® or(y))/(z <g> (z x ® z) <g> or(io) = 

(Z ® (Z x (8) 1/*) ® or(V))/(Z ® (Z x ® Z) ® or(V)) = 

Z g> (Z x <g> F*/Z) g> or(V) = 
l®End(V*/l) ®or(V). 

We have to quotient out the last space by Xi/(l <S> (Z -1 ® Z) Cg) or(V)). The 
result is 

Z® Endo(V*/l) ®or(V), (2.10) 

where .EncZo denotes the trace zero endomorphisms. 

Let us denote by N the bundle over P(V*) whose fiber over Z is equal to 
(12.1 Op . Clearly iV is GL(^/)-equivariant. Thus so far we obtained that the 
map % uniquely factorizes via a linear map 

H: C°°{F{V*),N) — + C oc (G'i(K), M) (2.11) 
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which is an injective GL(V) equivariant map. As we have mentioned previ- 
ously, Im(%) C H. Hence 

Im(U) C S. (2.12) 

It is easy to see that the composition 

SoU: C°°{F{V*),N) —^C 00 (Ox 

is a non-zero GL(V)-equivariant map. Since the target is an irreducible 
representation, the image of S o j-L is everywhere dense. Furthermore the 
Casselmann-Wallach theorem (see Theorem 13.91 below) implies immediately 
that S o % is onto, but we will not need this fact. 
Let us show that 

Im(H) HKerS ^ {0}. (2.13) 

A short proof of this claim was personally communicated to us by S. Sahi. 
We reproduce here his argument. 

Let us assume on the contrary that ImH D KerS = {0}. Then 

SoU: C°°{¥{V*),N) — ► C°°{G ri {V),L) 

is an isomorphism of the underlying Harish-Chandra modules (and in fact of 
the actual spaces by the Casselmann-Wallach theorem, though we will not 
need this fact here). We will get a contradiction by comparing the 50(3)- 
actions of them. 

Let us fix a Euclidean metric and an orientation on V. Then the space 
C°°(Gri(V), L) can be 5*0(3)-equivariantly identified with the space of smooth 
functions on RP 2 . It is well known that this space is a direct sum of irre- 
ducible representations of SO (3) with highest weights 2m, m G Z> , each 
entering with multiplicity 1 exactly. 

Now let us show that C°°(F(V*), N) contains some extra irreducible 
S'0(3)-representations. In the following computation the symbol "0" on the 
bottom denotes the traceless part. We have 

N\t = I g> End (V*/l) ® or(V) = 

SO(3) 

i®[(y*/iy®(y*/i)] ®ar(v) ~ 

/ (g) [{V*/Lf\ = 
I® A 2 (V*/l)®[Sym 2 (V*/l)} . 
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But I g> A 2 (V*/l) = A 3 V* ~ R. Thus we see that SO(3)-action on 
C°°(¥(V*), N) is a direct sum of the space of functions on P(V*) (which 
coincides with the previous space) and another infinite dimensional space. 
This implies ( 12. 13[) . 

Since S o R 7^ and Im(fl) H KerS 7^ 0, it follows that the length of 
C°°(¥(V*),N) as GL(y)-module is at least 2. The injectivity of U and f l2~T2l) 
imply 

2 < length(C°°(P(V r *), N)) < length(S). (2.14) 
But by Proposition 12.21 and Lemma 11.71 we obtain 

length(H) < 3. 

Consequently 

H(C°°(F(V*),N)) = E, 

and both have length 2. 

This implies Theorem 12.51 

3 Reminder on representation theory. 

We recall the Beilinson-Bernstein theorem on localization of g-modules fol- 
lowing [9], and the Casselman-Wallach theorem [18], |41j . 

Let G be a complex reductive algebraic group. Let T denote a Cartan 
subgroup of G. Let B be a Borel subgroup of G containing T. Let q denote 
the Lie algebra of G, t the Lie algebra of T, and b the Lie algebra of B. Let 
n denote the nilpotent radical of b. 

3.1 Remark. In our applications we will take G = GL 3 (C), T will be the 
subgroup of diagonal invertible matrices, B will be the subgroup of upper tri- 
angular invertible matrices, n is the Lie algebra of upper triangular matrices 
with zeros on the diagonal. 

Let R(t) C t* be the set of roots of t in g. The set R(i) is naturally divided 
into the set of roots whose root spaces are contained in n and its complement. 
The letter est is denoted by -R + (t); it is equal to the set of roots of t in g/b. If 
a is a root of t in g then the dimension of the corresponding root subspace g a 
is called the multiplicity of a. Let pt, be the half sum of the roots contained 
in R + (t) counted with their multiplicities. 
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For a root a let a v G t denote the corresponding coroot (see e.g. [20J, 
§2.4.2). We say that A G t* is dominant if for any root a G R + (t) we have 
< A, a v —1, —2, .... We shall say that A G t* is regular if for any root 
a G -R + (t) we have < A, a v >^ 0. 

For the definitions and basic properties of the sheaves of twisted dif- 
ferential operators we refer to [TH]. Here we will present only the explicit 
description of the sheaf V>\ in order to agree about the normalization. 

Let X be the full flag variety of G (then X = G/B). X can be identified 
with the variety of all Borel subalgebras of g. Let Ox denote the sheaf of 
regular functions on X. Let U(g) denote the universal enveloping algebra 
of g. Let U° be the sheaf U(g) ®c O x , and 0° := g ® c Ox- Let Tx be 
the tangent sheaf of X. We have the canonical morphism a : g° — > Tx- 
Let also b° := Kera = {£ G g°\ £ x G b^Vx G X}, where ^ C g denotes 
the Borel subalgebra corresponding to x G X. Let A: b — s> C be a linear 
functional which is trivial on n (thus A G t*). Since b x /[b x , b x ] are canonically 
isomorphic for different x G X, A defines a morphism A°: b° — > Ox- We 
will denote by T>\ the sheaf of twisted differential operators corresponding 
to A — pi,, i.e. T>\ is isomorphic to U°/Xx, where X A is the two sided ideal 
generated by the elements of the form 

e-(A-Pb)°(o, 

where ^ is a local section of b°. Let D\ := T(X, Pa) denote the ring of global 
sections of T>\. We have a canonical morphism U(g) — > D\. For the full 
flag variety X this map is onto ([9]). Let us also denote by V\ — mod (resp. 
D\ — mod) the category of T>\- (resp. -Da - ) modules. 

In this notation one has the following result due to Beilinson and Bern- 
stein [9]. 

3.2 Theorem (Beilinson-Bernstein) . (1) If X G t* is dominant then the 
functor of global sections T : T>\ — mod — > U(g) — mod is exact, namely F 
maps exact sequences to exact ones. 

(2) If X G t* is dominant and regular then the functor F is also fully 
faithful, namely F induces a bijection on the set of morphisms between any 
two objects ofT>\ — mod. 

3.3 Remark. Let us notice that when A = /?& we get the usual (untwisted) 
ring D and sheaf T> of differential operators. This A = pb is regular and 
dominant, and hence the Beilinson-Bernstein theorem is applicable. 
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Note also that the functor T can be considered as taking values in the 
category D\ — mod, rather than U(g) — mod. Considered in this way, T 
has a left adjoint functor, called the Beilinson-Bernstein localization functor, 
A : D\ — mod — > V x - mod. It is defined as A(M) =V x ®d x M. 

Part (1) of the next lemma was proved in [9], parts (2), (3) can be found 
in [TB], Proposition 1.6.6. 

3.4 Lemma. Suppose V : T>\ — mod — > D\ — mod is exact. Then 

(1) the localization functor A : D\ — mod — > T>\ — mod is the right 
inverse ofT: 

T o A = Id; 

(2) T sends simple objects to simple ones or to zero. 

(3) T sends distinct simple objects to distinct ones or to zero. 

Now let us discuss the Casselman-Wallach theorem. 

3.5 Definition. Let 7r be a continuous representation of a Lie group Go in a 
Frechet space F. A vector £ 6 F is called Go-smooth if the map g i— > 7r(#)£ 
is an infinitely different iable map from Go to F. 

It is well known (see e.g. Section 1.6) that the subset F sm of smooth 
vectors is a Go-invariant linear subspace dense in F. Moreover it has a natural 
topology of a Frechet space (which is stronger than the topology induced from 
F), and the representation of Go in F sm is continuous. Moreover all vectors 
in F sm are Go-smooth. 

Let G be a real reductive group. Assume that G can be imbedded into 
the group GLn{R) for some iV as a closed subgroup invariant under the 
transposition. Let us fix such an imbedding p : G H GLjy(R). (In our 
applications Go will be either GL n (R) or a direct product of several copies 
of GL n (M).) Let us introduce a norm | • | on G as follows: 

\g\:=m a x{\\p{ff)\\A\pi9- 1 )\\} 
where || • || denotes the usual operator norm in Mr. 

3.6 Definition. Let n be a smooth representation of Go in a Frechet space 
F (namely F sm = F). One says that this representation has moderate growth 
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if for each continuous semi-norm A on F there exists a continuous semi-norm 
v\ on F and d\ G K such that 

X(ir(g)v) < \g\ d *v x (v) 

for all g G G, v G F. 

The proof of the next lemma can be found in |41j . Lemmas 11.5.1 and 
11.5.2. 

3.7 Lemma, (i) If (tt,Gq,H) is a continuous representation of Go in a 
Banach space H , then (it, Gq, H sm ) has moderate growth. 

(ii) Let (tt, Gq, V) be a representation of moderate growth. Let W be a 
closed G -invariant subspace ofV. Then the representations of Gq in W and 
V/W have moderate growth. 

Recall that a continuous Frechet representation (n, Gq, F) is said to have 
finite length if there exists a finite filtration 

= F C F l C ■ ■ ■ C F m = F 

by Gq- invariant closed subspaces such that is irreducible for any i, 

i.e. it does not have proper closed Go-invariant subspaces. The sequence of 
all consecutive quotients 

Ft,F 2 / F\, ■ ■ ■ , F m /F m _i 

is called the Jordan-Holder series of the representation 7r. It is well known 
(and easy to see) that the Jordan-Holder series of a finite length representa- 
tion is unique up to a permutation. 

3.8 Definition. A Frechet representation (p, Gq, F) of a real reductive group 
Go is called admissible if its restriction to a maximal compact subgroup K 
of Go contains an isomorphism class of any irreducible representation of K 
with at most finite multiplicity. (Remind that a maximal compact subgroup 
of GL n {R) is the orthogonal group 0{n).) 

3.9 Theorem (Casselman-Wallach [18], [H]). Let Gq be a real reductive 
group. Let (p, Gq, Fx) and (tt,Gq,F 2 ) be smooth representations of moderate 
growth in Frechet spaces Fi,F 2 . Assume in addition that F 2 is admissible of 
finite length. Then any continuous morphism of GQ-modules f : Fi — > F 2 
has closed image. 
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4 Representations of GL^(M) 



Let P be the Borel subgroup GL 3 (M) consisting of upper triangular 3x3 
matrices. The positive roots are (— 1, 1, 0), (— 1, 0, 1), (0, — 1, 1). Hence the 
half sum of the positive roots is p = (—1, 0, 1). 

The main goal of this section is to prove Proposition 12.21 We will imme- 
diately see that Proposition 12.21 is equivalent to the following result. 

4.1 Theorem. Let ip(x,y, z) = l^l" 1 . Then the representation Indpip (the 
induction is not normalized!) has length three, and all the irreducible sub- 
quotients are pairwise non-isomorphic. 

The equivalence of Proposition 12.21 and Theorem 14 . 1 1 follows from an easy 
observation that the two induced representations considered in these state- 
ments are associate, and the well known fact (see e.g. [ID], Proposition 
4.1.20) that associate representations have the same Jordan- Holder series. 

To prove Theorem I4.1[ let us observe first that the character if> is dom- 
inant and regular. Let us denote by the corresponding sheaf of twisted 
differential operators on the flag manifold X. 

Now let us construct an 0(3, (C)-equivariant D^-module Ai^ on X such 
that the [/(g)-module of global sections T(X, Ai^p) coincides with the Harish- 
Chandra module of Indp^ip. Let U be the open 0(3, C)-orbit in X. Let 
j : U ^ X be the identity imbedding (which is an affine morphism). Fix a 
flag (E C F) G U. The category of 0(3, (C)-equivariant coherent P^-modules 
on U is equivalent to the category of finite dimensional representations of the 
group of connected components of the stabilizer of (E,F) in 0(3, C). This 
stabilizer is equal to 

0(1) x 0(1) x 0(1) ~ {±1} x {±1} x {±1}. 

Let us consider the representation of this group given by 

(A,B,C) ^ A. 

Let Ai^po be the corresponding 0(3, C)-equivariant P^-module on U. (It is 
easy to see that Ai^o is a locally free rank one Ou- module.) Define 

Mit, := j*M^ - 
It is not hard to see that Ai^ is the required P^-module. 



24 



By the Beilinson-Bernstein Theorem 13.21 and Lemma 13. 4[ Theorem 14.11 
is equivalent to saying that Ai^ has length three with non- isomorphic irre- 
ducible subquotients in the category of 0(3, (C)-equivariant ^-modules. 

The sheaf V>^, is the sheaf of differential operators acting on algebraic 
sections of the GL 3 (C)-equivariant line bundle £ over X which is defined as 
follows: for a complex flag (E C F) in C 3 , the fiber of £ over (E C F) is equal 
to E*. Then we have an equivalence of categories of 0(3, (C)-equivariant T>^- 
modules and of 0(3, (C)-equivariant T>- modules (here T> denotes the sheaf of 
usual untwisted differential operators on X) which is given by F i— > F ®o x 
£ _1 , where Ox denotes the sheaf of regular functions on X. 

Thus we have to prove the following proposition. 

4.2 Proposition. The 0(3, C)-equivariant T> -module 

M := ® 0x CT l (4.1) 

has length three and all the irreducible subquotients are pairwise non-isomorphic. 

Now let us describe explicitly the 0(3, C)-orbits on the variety of complete 
flags in C 3 . Let us denote by X the variety of complete flags in C 3 , namely 
the set of pairs (E, F) where E is a complex line, F is a complex 2-plane, 
and E C F. The following proposition is essentially well known and its proof 
is left to the reader as an (easy) exercise. We denote by B a given non- 
degenerate symmetric bilinear form on C 3 such that 0(3, C) is the group of 
linear transformations of C 3 preserving this form. 

4.3 Proposition. Two complete flags (E, F) and (E', F') in C 3 belong to the 
same 0(3, C) -orbit if and only if the following two conditions are satisfied: 

(i) ranks of the restrictions of B to E and to E' are equal; 

(ii) ranks of the restrictions of B to F and to F' are equal. 

4.4 Remark. It is easy to see that the rank of the restriction of B to any 
line in C 3 is either or 1, and the rank of the restriction of B to any plane 
in C 3 is either 1 or 2. 

Proposition 14.31 implies that the 0(3, C)-orbits on X are described by the 
following diagram: 
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Let us explain the notation used in this diagram. Here the pair of numbers 
corresponds to the orbit of flags (E C F) such that the restriction of 
the form B to E has kernel of dimension i, and the restriction of the form 
B to F has the kernel of dimension j. The arrows point from an orbit to 
another orbit contained in its closure. 

One can easily check the following lemma. 

4.5 Lemma. (1) The orbit (0,0) is open. 

(2) The codimensions of the orbits (0, 1) and (1,0) are equal to 1. 

(3) The codimension of the orbit (1, 1) is equal to 2. 

Let us start proving Proposition 14.21 by describing the module M. more 
explicitly. Let U be the open 0(3,C)-orbit in X, namely U corresponds to 
(0, 0) in the above diagram. Let j : U Hlbe the identity imbedding. Fix 
a flag (E,F) G U. The category of 0(3, (C)-equivariant coherent ©-modules 
on U is equivalent to the category of finite dimensional representations of the 
group of connected components of the stabilizer of (E,F) in 0(3, C). This 
stabilizer is isomorphic to the group 0(1) x 0(1) x 0(1). The representation 
of this group corresponding to the restriction Mq of M to U is equal to 
(A, B,C) A (recall that 0(1) = {+1, —1})- Then one can easily see that 

M=j*M , (4.2) 

where j* denotes the push-forward in the category of ©-modules. 

Let us now come back to our situation and let us describe the perverse 
sheaf corresponding to our D-module M.. Let us denote by M (resp. M ) 
the 0(3, C)-equivariant perverse sheaf on X (resp. U) corresponding to M. 
(resp. .Mo) via the Riemann-Hilbert correspondence (we refer to [TT] for 
this notion). Clearly M = j*Mo where j* is the push- forward in the derived 
category of sheaves; since j : U — > X is an affine open imbedding, perverse 
sheaves go to perverse sheaves (see [10]). It is clear that M is a local system 
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on U of rank one, shifted by 3 to the left. The following lemma describes the 
monodromies of M around the codimension one orbits (0, 1) and (1, 0). 



4.6 Lemma. The monodromy of M around the orbit (1,0) is equal to 
and its monodromy around the orbit (0, 1) is equal to 1. 



T; 



The proof is by a direct computation; it is left to the reader. Let us now 
discuss the local system M near the minimal orbit (1, 1). Let T denote a 
normal slice at some point on this orbit. It will be shown below that T is 
isomorphic to C 2 with the following stratification: 



T= SUPUQU{0}, 



(4.3) 



where P and Q are parabolas passing through the origin minus this origin 
and such that their closures P and Q are tangent to each other at 0. S 
is the open strata which is the complement of the two parabolas. P is the 
intersection of T with the orbit (0, 1), and Q is the intersection of T with 
the orbit (1,0). Let us describe the stratification (14. 3[) explicitly. 

Let us choose a basis ei, e2, e^ in C 3 so that the matrix B of our quadratic 
form in this basis is 

"001 
B= 1 
1 

Consider the flag x := (E C F ) defined by E = span{ei}, F = span{ei,e 2 }. 
Clearly, this flag belongs to the minimal orbit. By a straightforward compu- 
tation one easily checks that the Lie algebra of the stabilizer of xq in 0(3, C) 
consists of the matrices of the form 



Lie(St(x )) 



a b 









-b 
-a 



The transversal in Lie(0(3,C)) to this subalgebra can be chosen to be 
equal to 


-c 
c 

It also can be identified with the tangent space at Xo of the 0(3, C)-orbit of 

X . 
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Consider now the set of matrices of the form U 



the vectors 




a 
b a 



. Consider 



£i(a, b) = ei + Uei 



1 

a 
b 



,&(a,b) = e 2 + Ue 2 




1 

a 



Consider the flag X (a, b) := (E(a,b) C F (a, b)), where we denote E (a, b) = 
span{^i(a,b)} and F(a,b) = span{^i(a, b), £2(0, &)}■ If is eas y to see that 
the flags X(a, 6) with a,b E C, form a transversal to the minimal orbit; this 
transversal will be denoted by T. Let us now compute the intersection of 
this transversal with the other orbits. We have: 



B(Z 1 (a,b)M",b))=[ 



£(6MU 2 M))= [ lab] 



B(Ua,b),U*,b))=[0 1 a] 





' 





1 " 




" f " 


lab] 





1 







a 


1 










b 



' 





1 " 




" " 





1 







f 


1 










a 


" 





f ' 




' 





f 







1 


1 










a 



26 + a 2 , 



= 2a, 



= f. 



F(a, b) is equal to 



Thus we obtain that the matrix of the restriction of the quadratic form B to 
2b + a 2 2a 
2a 1 

Thus we see that the intersection of the transversal with the closure of 
the orbit (0, 1) consists of pairs P := {(a, 6)|26 = 3a 2 }, and the intersection 
of the transversal with the closure of the orbit (1,0) consists of the pairs 
Q := {(a, b)\2b = —a 2 }. In particular we see that P and Q are two different 
parabolas on C 2 with a single common point (0, 0) and which are tangent to 
each other at this point. To be consistent with the above notation, denote 
P:=P\{0},Q:=Q\{0}. 

Let us agree on the following notation: for any 0(3, C)-equivariant per- 
verse sheaf T on X or on a locally closed subset of X, we will denote by T 
the restriction of T to the intersection of the transversal T with this subset. 
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Then Lemma 14.61 implies that in this notation M is a rank one local sys- 
tem on S (up to a cohomological shift) with monodromy +1 around P, and 
monodromy —1 around Q. Let j : S — > T denote the identity imbedding. 
It is easy to see that the restriction j*M is equal to j*M . We will need the 
following lemma. 

4.7 Lemma. The "perverse sheaf j*M Q has length 3. Moreover the smooth 
parts of all the irreducible subquotients are supported on different strata, and 
they are rank one local systems (up to a cohomological shift). 

Before we prove Lemma 14 .7\ let us finish the proof of Theorem 14.11 It is 
easy to see that the functor of restriction to the transversal T is an exact and 
faithful functor from the category of 0(3, C)-equivariant perverse sheaves on 
X to the category of perverse sheaves on T (this is because T intersects every 
orbit). This and Lemma H2] imply that j*M has length at most 3 as 0(3, C)- 
equivariant perverse sheaf. Hence it remains to show that the restriction to 
T of any irreducible subquotient of j*M (as 0(3, C)-equivariant perverse 
sheaf) is an irreducible perverse sheaf; that clearly will imply that the length 
of j*M is equal to 3. 

Let N be an irreducible subquotient of j*M as an 0(3, C)-equivariant 
perverse sheaf. Consider its restriction N to T. We want to prove that N is 
irreducible. Assume in the contrary that it is not irreducible. By Lemma H~T1 
the smooth parts of different irreducible subquotients of N are supported on 
different strata, and all of these smooth parts are rank one local systems. 

Any irreducible 0(3, C)-equivariant perverse sheaf, can be constructed 
as follows. There exists a unique 0(3,C)-orbit O C X and an irreducible 
0(3, (C)-equivariant local system L on O such that N is isomorphic to the 
Goresky-Macpherson extension of L. It is easy to see that N is the Goresky- 
Macpherson extension of L. 

Let us show that rk(L) = 1. Assume in the contrary that rk(L) > 
2. Then N is not irreducible by Lemma 14. 7\ moreover there are at least 
rk(L) irreducible subquotients such that their smooth part is a rank one local 
system supported on OC\T. This contradicts Lemma W77\ according to which 
the smooth parts of all irreducible subquotients of N must be supported on 
different orbits. 

Thus L is a rank one local system. Its Goresky-Macpherson extension 
much be irreducible. Thus iV is irreducible. Thus we have shown that the 
restriction to T maps irreducible 0(3, C)-equivariant subquotients of j*M 
to irreducible perverse sheaves. Hence length of M = j*M is equal to 3. 
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Now it remain to prove Lemma 14.71 



Proof of Lemma 14.71 It will be more convenient to make change of 
variables on the transversal T as follows: 

z — a, 

I .a 2 

w = -( h b). 

2 K 2 J 

In coordinates (z, w) 

Q = { w = 0},P = {w = z 2 }. 

Recall that on S = T\(P U Q) we have a rank one local system M whose 
monodromy around P is equal to +1, and around Q to — 1. Let j : S — > 
T be the identity imbedding. We want to show that j*Mo is a length 3 
perverse sheaf, and the smooth parts of different irreducible subquotients are 
(appropriately shifted) rank one local systems with different supports. 

Let /: S — > T\Q be the identity imbedding, which is an affine mor- 
phism. Since the monodromy of M around P is equal to +1, there is a short 
exact sequence of perverse sheaves: 

K[2] — ►/,M — > jV[l], (4.4) 

where K is a rank one local system on T\Q, and N is a rank one local system 
on P(= P\{0}). 

Let g : T\Q — > T be the identity imbedding. It is an affine imbedding, 
hence g* is an exact functor between categories of perverse sheaves. Also 
3 — 9 ° f ■ Hence applying to (14.41) we get a short exact sequence of 
perverse sheaves on T 

g m K[2] — > j,M [2] j.N[l]. (4.5) 

It is easy to see that the monodromy of K around Q is equal to —1. Hence 
g*K[2] is irreducible perverse sheaf; clearly its smooth part is the rank one 
local system K[2] supported on T\Q. 

It remains to show that j*iV[l] has length 2, and smooth parts of the two 
irreducible subquotients are (shifted) rank one local systems on P and {0} 
respectively. Let us denote by h: P — > P the identity map. It is equivalent 
to prove that ft,*iV[l] has length 2, and the irreducible subquotients have 
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appropriate supports. It suffices to show that the monodromy of N around 
G P is equal to +1. To prove this, let us fix the loop around in P given 
by 

{(e ie ,e 2W )\0<6 <2tt}. (4.6) 
Around each point of this loop let us fix a small loop 

{(e ie ,e 2ie + 66^)10 <cu < 2tt}, (4.7) 

where < e < 1 is fixed, 9 G [0, 27r] is arbitrary. It is easy to see that each 
point on the loops (14.71) belongs to the open stratum S C T, and all these 
points together form a 2-dimensional real torus T 2 . For a fixed 8, the loop 
(14.71) is a loop in S around P. 

Let us write the torus T 2 as a product of circles T 2 = S\ x S2, where 
5*i corresponds to the parameter 6 in (14. 7p . and S2 to w. Let 7Ti : T 2 — > Si 
denote the natural projection. 

Let us denote by Mq the restriction of the (shifted by 2) local system Mq 
to the torus T 2 . Clearly the restriction of N[l] to the circle (14.61) is equal to 
R l n u (Mo) (where R 1 7iu is the first right derived functor of the push-forward 
under tti) which is a rank one local system on the circle Si. We want to show 
that its monodromy is equal to +1. 

Any local system on torus T 2 is characterized (up to isomorphism) by its 
monodromies along the meridians. The monodromy of M along the meridian 
S 2 is equal to +1. Then evidently the monodromy of -R^i^Mo is equal to 
the monodromy of Mq along the meridian Si, which we can choose to be 

{(e w ,e 2te + e)\6e [0,2tt]}. (4.8) 

Thus it remains to show that the monodromy of M along the circle (14.81) is 
equal to +1. To do that, let us go back from the torus to C 2 = T. In C 2 we 
can close the loop (14. 8 p by a disk D defined by 

D := {(z,z 2 + e)\ \z\ < 1}. 

Clearly dD equals the circle (14. 8p ; D does not intersect P; and does in- 
tersect Q in exactly two points (±iy/e, 0), in both of them the intersection 
is transversal. The latter implies that the monodromy of Mq around dD is 
equal to the product of monodromies of the two loops, each surrounding once 
exactly one of the the two points of intersection. But the monodromy of each 
such loop is equal to —1, hence their product is +1. Hence the monodromy 
of Mq around ( 14. 8p is +1. Hence the monodromy of iV around G P is +1. 
Lemma 14.71 is proved. 
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